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ERROR CONCEALMENT IN A VIDEO SIGNAL 



5 This invention relates to error concealment in a video signal and to a method 
and apparatus therefore. 

One of the recent targets in mobile telecommunications has been to increase 
the speed of the data transmission in order to enable incorporation of 

10 multimedia services to mobile networks. One of the key components of 
multimedia is digital video. Transmission of vtdeo comprises a continuous 
traffic of data representing moving pictures. As is generally known, the 
amount of data needed to transfer pictures is high compared to many other 
types of media, and so far usage of video in low bit-rate terminals has been 

15 negligible. However, significant progress has been achieved in the area of 
low bit-rate video compression. Acceptable video quality can be obtained at 
bit-rates around 20 kilo bits per second. As a result of this progressive 
reduction in bit-rate, it is expected that video is shortly going to become a 
viable service to offer over channels such as mobile channels. 

20 

A video sequence consists of a series of still images or frames. Video 
compression methods are based on reducing the redundancy and 
perceptually irrelevant parts of video sequences. The redundancy in video 
sequences can be categorised Into spatial, temporal and spectral redundancy. 

25 Spatial redundancy means the correlation between neighbouring pixels within 
a frame. Temporal redundancy means the correlation between areas of 
successive frames. Temporal redundancy arises from the likelihood of 
objects appearing In a previous image appearing in the cun-ent image too. 
Compression can be achieved by generating motion compensation data 

30 which describes the motion (i.e. displacement) between similar areas of the 
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current and a previous image. The current image Is tlius predicted from the 
previous one. Spectral redundancy means the correlation between the 
different colour components of the same image. 

However, sufficient compression cannot usually be achieved by just reducing 
the redundancy of the sequence. Thus, video encoders try to reduce the 
quality of those parts of the video sequence which are subjectively the least 
Important. In addition, the redundancy of the encoded bitstream is reduced 
by means of efficient lossless coding of compression parameters and 
coefficients. The main technique is to use variable length codes. 

Video compression methods typically differentiate images which can or 
cannot utilise temporal redundancy reduction. Compressed images which do 
not utilise temporal redundancy reduction methods are usually called INTRA 
or l-frames whereas temporally predicted images are called INTER or P- 
frames. In the INTER frame case, the predicted (motion-compensated) image 
is rarely precise enough and therefore a spatially compressed prediction error 
image is also associated with each INTER frame. 

Compressed video is easily corrupted by transmission errors, mainly for two 
reasons. Firstly, due to utilisation of temporal predictive differential coding 
(INTER frames), an error is propagated both spatially and temporally. In 
practice, this means that once an error occurs. It is easily visible to the human 
eye for a relatively long time. Especially susceptible are transmissions at low 
bit-rates where there are only a few INTRA-coded frames (the transmission of 
INTRA-coded frames would stop the temporal error propagation). Secondly, 
the use of variable length codes increases the susceptibility to errors. When a 
bit error alters the codeword to another one of different length, the decoder 
will lose codeword synchronisation and also decode subsequent error-free 
codewords (comprising several bits) incorrectly until the next synchronisation 
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code. (A synchronisation code is a bit pattern which cannot be generated 
from any legal combination of other codewords.) 

Every bit in a compressed video bitstream does not have an equal importance 
to the decompressed images. Some bits belong to segments defining vital 
information such as picture type (e.g. INTRA or INTER), quantiser value and 
optional coding modes that have been used. In H.263. the most vital 
information Is gathered in the picture header. A transmission en^r in the 
picture header typically causes a total misinterpretation of the subsequent bits 
defining the picture content. Due to utilisation of temporal predictive 
differential coding (INTER frames), the error is propagated both spatially and 
temporally. Thus, a normal approach to picture header corruption is to freeze 
the previous picture on the screen, to send an INTRA picture request to the 
transmitting terminal and to wait for the requested INTRA frame. This causes 
an annoying pause in the received video. 

Transmission errors have a different nature depending on the underlying 
network. In packet-switched networks, transmission errors are typically 
packet losses (due to congestion in network elements). In circuit-switched 
networks, transmission errors are typically bit errors where a "V is corrupted to 
'0' or vice versa. 

To impede degradations In images introduced by transmission enrors. 
retransmissions can be used, error detection and/or error correction methods 
can be applied, and/or effects from the received corrupted data can be 
concealed. Normally retransmission provides a reasonable way to protect 
video data streams from errors, but large round-trip delays associated with 
low bit-rate transmission and moderate or high error rates make It practically 
impossible to use retransmission, especially with real-time videophone 
applications. Error detection and correction methods usually require large 
overhead since they add some redundancy to the data. Consequently, for 



wo 00/27129 



PCT/EP99/08162 



10 



4 

low bit-rate applications, error concealment can be considered as a preferred 
way to protect and recover images from transmission errors. Video error 
concealment methods are typically applicable to transmission errors occurring 
through packet loss and bit corruption. 

H.263 is an ITU-T recommendation of video coding for low bit-rate 
communication which generally means data rates below 64 kbps. The 
recommendation specifies the bitstream syntax and the decoding of the 
bitstream. Currently, there are two versions of H.263. Version 1 consists of 
the core algorithm and four optional coding modes. H.263 version 2 is an 
extension of version 1 providing twelve new negotiable coding modes. 



Pictures are coded as luminance (Y) and two colour difference (chrominance) 
components (Cb and Cr). The chrominance pictures are sampled at half the 
15 resolution of the luminance picture along both co-ordinate axes. Picture data 
is coded on a block-by-block basis, each block representing 8x8 pixels of 
luminance or chrominance. 

Each coded picture, as well as the corresponding coded bitstream, is 
20 an-anged in a hierarchical structure with four layers, which are from bottom to 
top: block layer, macroblock layer, picture segment layer and picture layer. 
The picture segment layer can either be arranged as a group of blocks or a 
slice. 

25 Block layer data consists of uniformly quantised discrete cosine transform 
coefficients, which are scanned in zigzag order, processed with a run-length 
encoder and coded with variable length codes. 



30 



Each macroblock relates to 16 x 16 pixels of luminance and the spatially 
corresponding 8x8 pixels of chrominance components. In other words, a 
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macroblock consists of four 8x8 luminance blocl<s and tlie two spatially 
corresponding 8x8 colour difference blocks. Each INTER macroblock is 
associated witli a motion vector whicli defines the position of a corresponding 
area in the reference frame which resembles the pixels of the INTER 
5 macroblock. The INTER macroblock data comprises coded prediction error 
data for the pixels of the macroblock. 

Usually, each picture is divided into groups of blocks (GOBs). A group of 
blocks (GOB) typically comprises 33 macroblocks (arranged as 3 rows of 11 
10 macroblocks). Data for each GOB consists of an optional GOB header 
followed by data for the macroblocks within the GOB. 

If the optional slice stmctured mode is used, each picture is divided into slices 
instead of GOBs. A slice contains a number of consecutive macroblocks in 
1 5 scan-order. Data for each slice consists of a slice header followed by data for 
the macroblocks of the slice. 

The picture layer data contain parameters affecting the whole picture area 
and the decoding of the picture data. The coded parameter data is arranged 
20 in a so-called picture header. 

Picture and GOB (or slice) headers begin with a synchronisation code. No 
other code word or a legal combination of code words can form the same bit 
pattem as the synchronisation codes. Thus, the synchronisation codes can be 
25 used for bitstream error detection and for resynchronisation after bit errors. 
The more synchronisation codes that are added to the bitstream. the more 
error-robust the system becomes. 



30 



The Video Redundancy Coding (VRC) method has been introduced in several 
papers (e.g. Stephen Wenger. "Simulation Results for H.263+ Error 



wo 00/27129 



PCT/EP99/08I62 



6 

Resilience Modes K, R. N on the internet". ITU-T, SG16, Question 15. 
document Q15-D-17. 7-Apr-1998). Its objective is to provide graceful video 
quality degradation against packet losses in packet-switched networks. The 
following paragraphs explain the basics of the method. 
5 . 

The principle of the VRC method is to divide the sequence of pictures into two 
or more signals (or threads) in such a way that all frames are assigned to one 
of the threads in an interleaved fashion to form subsets of frames. Each 
thread (or subset of frames) is coded independently. Obviously, the frame- 

10 rate within one signal is much lower than the overall frame rate: half in the 
case of two threads (signals), a third in the case of three threads and so on. 
This may result in a substantial coding penalty because of the generally larger 
changes and the longer motion vectors typically required to represent 
accurately the motion related changes, between two INTER-frames within a 

15 signal or thread. At regular intervals, all the signals converge into a so-called 
Sync frame. From this Sync frame, a new series of threads is started. 

If one of the threads containing a subset of frames is damaged because of. 
say, a packet loss, the remaining threads stay intact and can be used to 
20 predict the next Sync frame. It is possible to continue the decoding of the 
damaged signal, which leads to slight picture degradation, or to stop its 
decoding which leads to a drop of the frame rate. If the size of the subsets is 
kept reasonably small, however, degradation will persist only for a very short 
time, until the next Sync frame is reached. 

25 

The decoder selects a Sync frame from one of the undamaged threads to 
decode the Sync frame. This means that the number of transmitted l-pictures 
can be kept small, because there is no need for complete re-synchronisation. 
If all threads are damaged between two Sync frames, it is not possible to 
30 accurately predict a Sync frame. In this situation, annoying artifacts will be 
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present until the next l-picture is decoded correctly, as would be the case if 
VRC were not employed. 

Currently. Video Redundancy Coding can be used with ITU-T H.263 video 
5 coding standard (version 2) if the optional Reference Picture Selection mode 
(Annex N) is enabled. However, there are no major obstacles to incorporating 
Video Redundancy Coding into other video compression methods too. 

Most known error concealment techniques are based on spatial and temporal 
10 interpolation schemes. Spatial interpolation is used in INTRA frames and 
INTRA-coded areas of INTER frames. Spatial interpolation means that lost 
areas are interpolated from spatially neighbouring areas. This can be done 
for example using the distance weighted average of the boundary pixels. 

15 Error concealment using temporal interpolation is more often used in low bit- 
rate video coding, since the number of INTRA frames is usually rather low. A 
very basic temporal interpolation scheme copies the lost areas from the same 
positions of the previous frame, i.e.. it treats the lost blocks as "not coded" 
blocks. In more advanced schemes, motion compensation is performed using 

20 either the median or average of the motion vectors of spatially neighbouring 
blocks. There have also been some proposals to use boundary pixel 
matching to find best motion vectors for the lost block. 

In low resolutions and at low bit-rates, the correlation between spatially 
25 neighbouring blocks is often rather low. Thus interpolated motion vectors 
based on spatially neighbouring pixel data may be far from the original values. 
This means that one-directional concealment schemes often fail to reconstruct 
the original blocks. Furthermore, if only motion vectors are used for 
concealment without even trying to recover the prediction en-or blocks, the 
30 picture becomes blurred, since a great amount of detail will be lost. In 
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practice, using current concealnnent schemes, errors or incorrectly concealed 
blocks are visible for a relatively long time. 

Previously proposed utilisation of VRC in error-prone environments suffers 
5 from a few problems. First, if the interval between Sync frames is short (often 
the thread length has been proposed to be 5 frames), compression efficiency 
is compromised. On the other hand, if the threads are longer, error 
concealment tends not to be effective and picture quality is compromised. 

1 0 In accordance with a first aspect of the invention there is provided a method of 
concealing an error in a frame of a video sequence, said video sequence 
comprising a plurality of frames and being encoded as at least two 
independently-coded signals, each of which represents a sub-set of frames of 
the video sequence, the method comprising receiving data representing a 

15 frame of the video sequence, identifying an error in the frame and concealing 
the enxjr by predicting corresponding data using at least one frame which is 
encoded in a signal other than that in which the error is identified. 

Thus the invention relates to a multi-threaded video coding scheme in which 
20 an en-oneous area is concealed by interpolating temporally and preferably 
bidirectionally from uncorrupted frames of another VRC thread. 

The invention provides means to achieve better image error concealment than 
prior-art solutions. Compared to prior-art Video Redundancy Coding 
25 methods, it makes it possible to use longer threads between Sync frames, 
thus increasing the compression efficiency. 

Preferably the corresponding data is predicted bidirectionally i.e. using frames 
which occur in the video sequence previous and subsequent to the frame in 
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Which the error is identified, said previous and subsequent frames being 
encoded in at least one signal other than that in which the error is identified. 

Most advantageously, the corresponding data may be predicted using frames 
which occur In the video sequence immediately previous and/or subsequent 
to the frame in which the error is identified, said previous and subsequent 
frames being present in at least one other signal. 

The error may be initially concealed by predicting the corresponding data from 
a frame occumng previous to the said frame in the same signal as the frame 
in which the error is identified. 

When the encoded signals include header information, a frame having an 
error in the header may be reconstructed by identifying an error in the header 
of a frame, determining whether the frame is an interframe-coded frame and, 
if so, predicting corresponding data using at least one frame which is encoded 
in a signal other than that in which the error is identified. 

According to a second aspect of the invention a method of video 
decompression comprises receiving at least two signals representing a video 
sequence, said video sequence comprising a plurality of frames, each signal 
representing a sub-set of frames of the video sequence, identifying an error in 
a frame of the video sequence, concealing the error in the frame by predicting 
corresponding data using at least one frame which is encoded in a signal 
other than that in which the error is identified, and displaying the frames of the 
video sequence. 

A third aspect of the invention relates to video error concealment apparatus 
for concealing an error in a frame of a video sequence, said video sequence 
comprising a plurality of frames and being encoded as at least two 
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independently-coded signals, each of which represents a sub-set of frames of 
the video sequence, the apparatus comprising: an input for receiving said at 
least two signals, identifying an error in a frame of the video sequence, and 
means for concealing the error by predicting corresponding data using at least 
5 one frame which is encoded in a signal other than that in which the error is 
identified. 

Preferably the concealing means is arranged to predict the corresponding 
data using frames which occur in the video sequence previous and 
10 subsequent to the frame in which the error is identified, said previous and 
subsequent frames being encoded in at least one signal other than that in 
which the error is identified. 

Most advantageously the concealing means is arranged to predict the 
15 con-esponding data using frames which occur in the video sequence 
immediately previous and/or subsequent to the frame in which the error is 
identified, said previous and subsequent frames being present in at least one 
other signal. 

20 The concealing means may be arranged to initially conceal the error by 
predicting the corresponding data from a frame occurring previous to the said 
frame in the same signal as the frame in which the en-or is identified. 

When the encoded signal includes header information, the apparatus may 
25 further comprise means for identifying an error in the header of a frame and 
means for determining whether the frame is an interframe-coded frame, 
wherein the concealing means is arranged to predict corresponding data 
using at least one frame which is encoded in a signal other than that in which 
the error is identified. 
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The invention will now be described by way of example only with reference to 
the acconnpanying drawings, in which: 

Figure 1 illustrates a nnulti-layer video coding scheme with two threads (or 
signals) and 3 frames per thread; 

Figure 2 illustrates the prior art operation of VRC when one of the two threads 
is damaged; 

Figure 3 illustrates one embodiment of the method according to the invention; 

Figure 4 illustrates a second embodiment of the method according to the 
invention; 

Figure 5 illustrates an error prediction method which may be used to conceal 
errors; 

Figure 6 shows an example of INTER picture header concealment; 

Figure 7 shows an example of the invention applied to a multi-threaded signal 
having three threads; and 

Figure 8 shows an embodiment of a decoder in accordance with the invention. 

According to a multi-threaded video coding scheme such as VRC, a video 

sequence comprising a plurality of frames 0, 1, 2, 3. n is encoded as at 

least two threads or signals, each of which represents a subset of frames of 
the video sequence. Each of the threads is encoded independently of the 
others and each of the frames of a thread are coded in an INTER-frame 
manner (although areas of a frame may be coded in an INTRA-frame manner 
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as is well known). Thus at least two threads of INTER-frames are coded and 
transmitted independently. Each of the INTER-frames (or P-frames) depends 
on an earlier INTER-frame of the thread but not on any information within the 
other threads. Frames to be coded are assigned to a thread in an interleaved 
5 manner. All threads start from a Sync frame, which may be an INTRA-frame 
although this is not essential. 

In the example shown in Figure 1. the video sequence comprises 2 threads 
10a, 10b each of which has three frames 12 per thread. Since only 2 threads 

10 are generated, each thread comprises alternate frames of the video 
sequence. It is to be noted that each thread 10a, 10b converges to a Sync 
frame 14. In VRC it is intended that the last frame in each thread (i.e. frame 5 
of thread 10a and frame 6 of thread 10b) independently generates the same 
Sync frame 14. Whilst in practice this appears difficult to achieve, simulations 

15 show that it is possible to come close enough (see the above paper by 
Stephan Wenger). 

Figure 2 illustrates the prior art operation of VRC when one of the two threads 
or signals is damaged. It has been proposed that if a thread is damaged, the 

20 thread is ignored and no longer used to generate the decompressed video 
sequence. The remaining intact threads are used to generate the video 
sequence. Say that frame P4 is damaged. The thread 10b is either no longer 
used to generate the decoded video sequence (which leads to degradation in 
the frame rate of the resulting video sequence) or the next frame P6 of the 

25 thread 10b is decoded with reference to the corrupted frame P4 which means 
that the error is propagated. 

Examples of concealment algorithms in accordance with the invention will 
now described. As in Figures 1 and 2, the examples show two threads (or 
30 signals) generated with 3 frames per thread. Assume that there are no 
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transmission errors in the first Sync frame nor in INTER-frames P1, P2. and 
P3. These frames are thus decoded and displayed normally. Assume that 
the decoder detects that some parts (but not all) of frame P4 are corrupted. 
The video decoder then starts to conceal the frame P4 with reference to a 
5 frame or frames of the uncorrupted thread 10a, The concealment can be 
done in conjunction with the decoding of the subsequent frame in the 
damaged signal. 

Figure 3 illustrates the operation of a first embodiment of the invention. 

10 Suppose that there is a transmission error in the compressed data for frame 
P4 (marked dP4 in the figure). This frame is decoded but the corrupted frame 
(dP4) is not usually displayed. However the corrupted areas of frame dP4 are 
concealed by using a uni-directional interpolation scheme utilising frame P3 of 
the uncorrupted thread 10a. Let us call the concealed frame P4*. Then error- 

15 free frame P5 of the undamaged thread 10a is decoded and displayed 
normally. Preferably, further concealment is then carried out by bidirectional 
concealment of frame P4 with reference to frames P3 and P5 of the 
undamaged thread. 

20 An example of an interpolation algorithm which may be used is discussed 
further below although other algorithms may be used. 

Figure 4 illustrates the operation of a second embodiment of the invention. 
Suppose that there is a transmission error in the compressed data for frame 

25 P4 (marked dP4 in the figure). This frame is decoded but the corrupted frame 
(dP4) is not usually displayed. Then error-free frame P5 of the undamaged 
thread 10a is decoded and displayed normally. After that, according to the 
invention, the corrupted areas of frame dP4 are concealed by using a 
bidirectional interpolation scheme utilising frames P3 and P5 of uncorrupted 

30 thread 10a. Let us call the concealed frame P4'. The advantage of using 
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bidirectional interpolation from an undamaged thread (as illustrated in Figure 
4) is preferred over uni-directional interpolation (as shown in Figure 3) since 
bidirectional interpolation generally gives better concealment results. 



5 In a third embodiment of the invention, the corrupted frame P4 is decoded 
and the corrupted areas are initially concealed using prior-art concealment 
methods using the P2 frame. The concealed frame (P4c) is displayed. Then, 
error-free frame P5 of the undamaged thread 10a is decoded and displayed 
normally. The concealed areas of frame P4^ are then refined by using a 

10 bidirectional interpolation scheme utilising frames P3 and P5 of uncorrupted 
thread 10a. Let us call the refined concealed frame P4". When decoding P6, 
frame P4" is used as the reference frame. Since P4" is likely to have better 
quality than initially concealed frame P4^, the quality of decoded frame P6 
(and any subsequent frame in the same temporal prediction thread) is likely to 

15 be better too. Then, the decoding process continues normally. 

Error concealment can be applied to recover video packets which have been 
lost or which contain CRC errors (if the transport layer is able to provide CRC 
error indications). In practice, only the blocks which are referenced in P6 
20 need to be concealed, unless frame P4 is referenced later on (the 
specification of the Reference Picture Selection mode of H.263 allows this). 



In its basic form, bidirectional interpolation for a two-threaded signal is 
performed as follows. The motion vectors in P5 are divided using the 
25 temporal references of the frames P3, P4, and P5 to get forward and 
backward motion vectors for the frame P4, i.e., fonA/ard motion vectors MVf 
are calculated component-wise (x and y components) by: 
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15 



15 

where TR^ stands for Time Reference of a frame n and MVg is a motion vector 
component from frame 3 to 5, Backward motion vectors MVb are calculated 
in the same way, but the numerator of the scaling factor is then TR5-TR4 and 
the sign is opposite to the original sign. The corrupted areas are then 
predicted from the two temporal directions and the final reconstructed block is 
a weighted average of the prediction blocks. Again, the time references are 
used as weighting factors. In mathematical terms, pixel (ij) of a block at 
frame P4 [denoted as PiX4(i j) ] is constructed using the following formula: 



(TR, ~ ri? J * Pix, (z + MVb, , J + MVb^ ) 
+ TR^-TR^ ~~ 

For simplicity this method assumes that the motion vector from P5 is directly 
applicable In the corresponding macroblock position in P4, i.e. motion within 
the area affected by a certain motion vector is not taken into account. This 
assumption is made purely to keep this example simple. 



Figure 5 illustrates an exemplary bidirectional interpolation scheme. It 
represents a very much simplified situation, where the black bar 30 in frame 
PS has moved to the right in frame P5 and the colour has lightened (hence 
the prediction error is nonzero). The subscript denotes the position of the 

20 black bar 30 in frame n. When constructing the frame P4" using bidirectional 
interpolation, motion vector MVf points to the position (3O3) where the bar was 
in the frame P3, and motion vector MVb points to the bar position (3O5) in the 
frame P5. The resulting bar (3O4) has a colour which is the weighted average 
of the reference bars, and its location is also a weighted average of the 

25 reference locations. Note that, for simplicity, this example is not block based. 



To further improve the algorithm, the following refinements can be made: 
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1. The frames P3, P4" (or P4*) and P5 are compared block by block inside the 
corrupted area. If blocks at spatially identical positions in P3 and P5 are the 
same (the block in P5 may have been transmitted as a "not coded" block), but 
the block is different in P4* (or P4"), the block is most probably corrupted. If 

5 the error has not been detected previously, it needs to be concealed. The 
concealment can be done by copying the corresponding block from P5, which 
produces a similar result to bidirectional interpolation, but is simpler. 

2. Any blocks which have been coded in INTRA mode in frame P5 should be 
10 copied to P4' (or P4") directly from P5, since bidirectional interpolation might 

cause unwanted anomalies. 

3. If there are some changes between P3 and P5, which have actually 
happened in the time period between frames P4 and P5, these changes 

15 should not be reflected in the concealed frame P4". In this case, uni- 
directional interpolation from P3 (i.e. backward prediction) produces better 
results. This kind of situation can be detected by comparing the changes in 
error free areas of P4, for example by performing bidirectional interpolation 
also for the error-free areas and checking which direction produces the more 

20 acceptable results. Furthermore, an additional weighting factor can be 
derived to give more weight to either frame P3 or P5, based on the test 
interpolation performed on the error-free areas. 

4. Bidirectional interpolation itself can also be further improved by fine tuning 
25 the motion vectors using for example some prior-art solutions, such as 

boundary pixel matching. This naturally requires that there are error-free 
blocks available in P4 to be used in the matching process. 

If the frame P4 is completely lost, refinements 3 and 4 cannot be used. 

30 
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So far error concealment has been discussed with reference to errors in the 
picture information of the video sequence i.e. in the data that is to be 
displayed. The invention is also applicable to frames which include errors 
occurring in the header of the video data. Conventionally such frames would 
5 not be recoverable and an INTRA-frame would be requested by the decoder 

Figure 6 illustrates a further embodiment of the invention. As in the other 
examples, there are two independent VRC threads. Assume that the picture 
header of P2 has been corrupted in the transmission channel and the decoder 

10 detects the corruption. Instead of immediately posting an INTRA frame 
request the decoder waits for the next frame (P3). P3 is interframe coded 
with respect to frame PI. The decoder therefore determines that the 
corrupted frame P2 is surrounded by frames belonging to another VRC thread 
and that P3 is temporally predicted from P1 . Thus, it is likely that P2 is also 

15 an INTER frame and that an estimation of P2 can be generated using a 
concealment method as described above with reference to frame PI and/or 
P3 of the uncorrupted thread 10a. The decoding then continues as normal. 

If the back-channel messaging feature of Reference Picture Selection mode 
20 of H.263 is in use, the decoder can signal that the entire P4 image is 
corrupted and that P3 should be used as a new reference frame for coding. 
The transmitting encoder receives this back-channel message and codes the 
next frame belonging to the corrupted thread (P6) using correctly decoded P3 
as a reference. In this way, the decoded stream is totally recovered without 
25 the need for the transmission of an INTRA frame. If back-channel messaging 
cannot be used, the decoded stream is totally recovered when the next VRC 
Sync frame arrives. Meanwhile the initial reference for temporal prediction in 
the corrupted VRC thread is the reconstructed frame P2. 
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Thus the invention means that longer threads may be used since INTRA 
frames are unlikely to be requested so frequently to recover the picture data. 
The compression efficiency of the multi-layer approach of video redundancy 
coding is therefore maintained without compromising the resulting picture 
5 quality. 

The examples have all been described with reference to two threads having 
three frames between each Sync frame. However it will be apparent to the 
reader that the invention is also applicable to embodiments having more than 

10 two threads and to threads having more or less than three frames. For 
instance, Figure 7 shows a video signal encoded as three threads 10a, 10b 
and 10c. each thread having three frames 12. Say that frame P5 of thread 
10b is corrupted. Errors in the corrupted frame P5 may be concealed by 
interpolation from frame P4 in the uncorrupted thread 10a and/or from frame 

15 P6 in the uncorrupted thread 10c. 

For the instance where the errors are concealed by bidirectional prediction 
from frames P4 and P6 the forward motion vectors are calculated as follows: 

TR — T'P 

20 MVf,=^ ' * *MV^,^ 

TRy-TR^ 

where TR^ stands for Time Reference of a frame n and MVy^ is the motion 
vector component from frame P7 to P4 of ttiread 10a. Bacl<ward motion 
vectors MVb are calculated in the same w^ay, but from the other uncorrupted 
25 thread 10c i.e.: 

' TR^-TR3 
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The corrupted areas are then predicted from the two temporal directions and 
the final reconstructed block is a weighted average of the prediction blocks. 
Again, the time references are used as weighting factors. 

5 Again, for simplicity, this method assumes that the motion vectors from P4 
and P6 are directly applicable in the corresponding macroblock position in P4, 
i.e. motion within the area affected by a certain motion vector is not taken into 
account. This assumption is made purely to keep this example simple. 

10 Although the Figures show the threads converging to a single Sync frame 14 
it should be understood that a Sync frame is coded for each thread 10a, 10b, 
10c etc. Hence, in Figure 7 for example: thread 10a comprises frames 0, 1, 4. 
7, 10, 11, 14, 17, 20; thread 10b comprises frames 0, 2, 5, 8, 10, 12, 15, 18 
etc.; and thread 10c comprises frames 0, 3, 6, 9, 10, 13, 16, 19 etc; frames 0, 

15 10, 20 etc. being Sync frames 14. The decoder however needs to decode 
only one occurrence of each Sync frame 14. 

Figure 8 shows an example of a video decoder in accordance with the 
invention. The decoder shown is arranged to decode a video sequence which 

20 has been encoded as two threads. The decoder comprises a buffer 30 for 
buffering incoming signals. A VLC decoder 32 decodes the incoming signals 
to retrieve quantised DCT coefficients of the signal which are output to an 
inverse quantiser 34 and also decodes the motion vector information which is 
output to motion compensator 35. An inverse DCT transformer 36 carries out 

25 an inverse transforms on the de-quantised DCT coefficients from the inverse 
quantiser 34 and outputs prediction data. The motion compensator 35 alters 
the picture data of a previous frame in accordance with the motion vectors. 
The pixel data output from transformer 36 is added to the picture data of a 
previous frame from the motion compensator 35 by an adder 38. The 

30 resulting picture data is then output to a buffer 39 for subsequent display. 
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The picture data output from the adder 38 is also stored in a set of frame 
stores 40a or 40b. The set of stores selected by the switch 42 is determined 
by the thread being decoded i.e. set 40a stores frames from thread 10a and 
5 thread 40b stores frames from thread 10b. The motion vectors are also 
stored in the frame stores 40a, 40b for use in error concealment. 

The motion compensator 35 selects a frame from a set of the frame stores, 
the set being determined by the frame being decoded i.e. if a frame of thread 
10 10 b is being decoded, the previous frame stored in set 40b is input to the 
motion compensator 35. 

An error detector 44 detects errors In the decoded video data. If, during 
decoding of a frame by the VLC decoder 32, an error is detected, the decoder 

15 initiates an error concealment process as described previously. The output of 
the error detector 44, which was previously connected to the output buffer 39, 
is switched to connect the output to an error concealment section 46. The 
error concealment section 46 carries out error concealment on corrupted parts 
of the current frame with reference to a frame or frames from another 

20 uncorrupted thread, as described above. The reconstructed frame is then 
output to the output buffer 39 for subsequent display. The reconstructed 
frame is also stored in the appropriate place in the appropriate set of frame 
stores 40a or 40b as determined by the switch 42 controlled by the processor 
(not shown) of the decoder. 

25 

The locations of the corrupted areas of the current frame under consideration 
are also stored in the frame stores 40a, 40b. 

The number of frame stores in each set of frame stores 40a, 40b is usually 
30 two to allow for en^or concealment in both the forward and the reverse 
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direction from an immediately, previous and subsequent frame respectively. 
However each set of frame stores may comprise a single frame store if only 
uni-directional prediction is to be used in error concealment. 



5 



Clearly the decoder may be provided with more than two sets of frame stores 
so that a multi-threaded signal having more than two threads may be 
decoded. 
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Claims 

1- A method of concealing an error in a frame of a video sequence, said 
video sequence comprising a plurality of frames and being encoded as at 
5 least tv\/o independently-coded signals, each of which represents a sub-set of 
frames of the video sequence, the method comprising: 

receiving data representing a frame of the video sequence and 
identifying an error in the frame 

concealing the error by predicting corresponding data using at least 
10 one frame which is encoded in a signal other than that in which the error is 
identified. 

2. A method according to claim 1 further comprising predicting the 
corresponding data using frames which occur in the video sequence previous 

15 and subsequent to the frame in which the enror is identified, said previous and 
subsequent frames being encoded in at least one signal other than that in 
which the error is identified. 

3. A method according to claim 2 further comprising predicting the 
20 corresponding data using frames which occur in the video sequence 

immediately previous and subsequent to the frame in which the en'or is 
identified, said immediately previous and subsequent frames being encoded 
in at least one signal other than that in which the error is identified. 

25 4. A method according to claim 3 wherein the step of identifying an error 
comprises comparing areas of the frame with corresponding areas of the 
immediately previous and subsequent frames and, if areas at spatially 
identical positions in the previous and subsequent frames are the same but 
the area is different in the frame under consideration, the area is deemed to 

30 be corrupted. 
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5. A method according to claim 4 wherein the error is concealed by 
copying the corresponding area from the subsequent frame. 

5 6. A method according to any of claims 2 to 5 wherein any areas which 
have been coded in INTRA mode in the subsequent frame are copied directly 
from the subsequent frame to the corresponding areas of the frame under 
consideration. 

10 7. A method according to any preceding claim further comprising initially 
concealing the error by predicting the corresponding data from a frame 
occurring previous to the said frame in the same signal as the frame in which 
the error is identified. 

15 8. A method according to any preceding claim in which the encoded 
signals include header information, the method further comprising identifying 
an error in the header of a frame, determining whether the frame is an 
interframe-coded frame and, if so, predicting corresponding data using at 
least one frame which is encoded in a signal other than that in which the error 

20 is identified- 

9. A method according to any preceding claim wherein the step of 
concealing the error involves performing bidirectional prediction for error-free 
areas of the frame under consideration and determining which direction of 

25 prediction produces the more acceptable results. 

10. A method according to claim 9 further comprising deriving a weighting 
factor to give more weight to either fonA/ard or reverse prediction, based on 
the test prediction performed on the error-free areas. 



30 
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11. A method of video decompression comprising 

receiving at least two signals representing a video sequence, said 
video sequence comprising a plurality of frames, each signal representing a 
sub-set of frames of the video sequence, 
5 decoding the frames of the received signals to produce data 

representing frames of the video sequence, 

identifying an error in a frame of the video sequence, 

concealing the error in the frame by predicting corresponding data 
using at least one frame which is encoded in a signal other than that in which 
10 the error is identified, and 

displaying the frames of the video sequence. 

12. Video error concealment apparatus for concealing an error in a frame 
of a video sequence, said video sequence comprising a plurality of frames 

15 and being encoded as at least two independently-coded signals, each of 
which represents a sub-set of frames of the video sequence, the apparatus 
comprising: 

an input for receiving said at least two signals, 

means for identifying an error in a frame of the video sequence, and 
20 means for concealing the error by predicting corresponding data using 

at least one frame which is encoded in a signal other than that in which the 
error is identified. 

13. Apparatus according to claim 12 wherein the concealing means is 
25 arranged to predict the corresponding data using frames which occur in the 

video sequence previous and subsequent to the frame in which the error is 
identified, said previous and subsequent frames being encoded in at least one 
signal other than that in which the error is identified. 
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14. Apparatus according to claim 13 wherein the concealing means is 
arranged to predict the corresponding data using frames which occur in the 
video sequence immediately previous and subsequent to the frame in which 
the error is identified, said previous and subsequent frames being present in 

5 at least one other signal. 

15. Apparatus according to claim 12, 13 or 14 wherein the concealing 
means is arranged to initially conceal the error by predicting the 
corresponding data from a frame occurring previous to the said frame in the 

10 same signal as the frame in which the error is identified. 

16. Apparatus according to any of claims 12 to 15 wherein each encoded 
signal includes header information, the apparatus further comprising means 
for identifying an error in the header of a frame and means for determining 

15 whether the frame is an interframe-coded frame, wherein the concealing 
means is arranged to predict corresponding data using at least one frame 
which is encoded in a signal other than that in which the error is identified. 
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